{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1e83c3fe",
   "metadata": {},
   "source": [
    "<br><font face=\"Times New Roman\" size=5><div dir=ltr align=center>\n",
    "<font color=blue size=8>\n",
    "    Introduction to Machine Learning <br>\n",
    "<font color=red size=5>\n",
    "    Sharif University of Technology - Computer Engineering Department <br>\n",
    "    Fall 2022<br> <br>\n",
    "<font color=black size=6>\n",
    "    Homework 2: Practical - SVM\n",
    "    <br>\n",
    "<font color=black size=4>\n",
    "    Alireza Gargoori Motlagh, Alireza Heidari\n",
    "    \n",
    "<br><br>\n",
    "<font size=4>\n",
    "   **Name**: <br>\n",
    "   **Student ID**: <br> <br>\n",
    "In this homework, we are going to implement the algorithms from the second and third chapters of the course, so reading the notebooks of these chapters would be quite useful. Also, we are going to evaulate and compare our models by the metrics defined in chapter 3. Keep in mind to complete all of the following questions and write your own codes in the TODO parts."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40a02596",
   "metadata": {},
   "source": [
    "## <font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "## Problem 3: Face Classification by SVM on Eigenfaces (150 points)\n",
    "In the following problem, we are going to build a classifier to distinguish the faces of 40 people on a toy dataset. The dataset includes 400 pictures of 40 people faces, each by a 64*64 pixel picture. Use the following cell to download the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "083b13eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "faces = datasets.fetch_olivetti_faces()\n",
    "print(faces.data.shape)\n",
    "print(faces.images[0].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "164fd336",
   "metadata": {},
   "source": [
    "<font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "### Visualization of the Faces (5 points)\n",
    "Plot an image of each class in one figure using subplots. (Note that the design matrix X is availabe in <code>faces.data</code> and the target values available in <code>faces.target</code> .)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7a0194c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec19e0b4",
   "metadata": {},
   "source": [
    "<font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "### Train-Test Split (2 points)\n",
    "Split the data into train and test sets and print the dimensions of each set. (70% for train and 30% for test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd164d8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4754cec8",
   "metadata": {},
   "source": [
    "<font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "### Dimensionality Reduction (18 points)\n",
    "As you might have guessed, the number of featuers, i.e. pixels, is quite large and training a classifier might take a considerbale time. However, using a wise method to reduce the dimension of our data could decrease the cost of training the model. In the following cell use PCA (Principal Component Analysis) to reduce the dimension of the dataset. Choose an appropirate number of components in your PCA and explain your reason for your choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93689523",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71ecd4e3",
   "metadata": {},
   "source": [
    "<font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "Plot the scree plot of the proportion of the variance explained(PVE) vs principal components to determine the number of principal components for futher analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aca4b9d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65c7d073",
   "metadata": {},
   "source": [
    "<font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "### Eigenfaces: Principal Components Visualization (25 points)\n",
    "Plot the mean face using your PCA model with <code>pca.mean_</code> attribute followed by a reshape to original dimensions. What is this mean vector?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31256083",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7c3b9a8",
   "metadata": {},
   "source": [
    "<font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "Eigenfaces is the name given to a set of eigenvectors when used in the computer vision problem of human face recognition. Plot the 30 first eigenfaces sorted by their rank from the most important to the lowest. Can you explain what these faces indicate?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a33ad8e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49d3feb2",
   "metadata": {},
   "source": [
    "<font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "Transform your train and test sets to the new feature space using your PCA module. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18b3ce57",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "734a5251",
   "metadata": {},
   "source": [
    "<font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "### Training your SVM Classifier (50 points)\n",
    "Train an SVM classifier on the transformed dataset in the new feature space. You should use Grid Search Cross Validation or other validation techniques to obtain the best hyperparameters for your classifier. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5c4097e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import svm\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "#TODO\n",
    "\"\"\"\n",
    "clf = svm.SVC(kernel='', C=?, degree=?, gamma=?)\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e49bd46",
   "metadata": {},
   "source": [
    "<font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "After training your model, we can check its prediction on test samples. Run the following cell to check your model predictions with the target values on test samples. Predictions with red label indicate false classification of your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aaf82502",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(15, 10))\n",
    "for i in range(30):\n",
    "    ax = fig.add_subplot(6, 5, i + 1, xticks=[], yticks=[])\n",
    "    ax.imshow(X_test[i].reshape(faces.images[0].shape),\n",
    "              cmap=plt.cm.bone)\n",
    "    y_pred = clf.predict(X_test_new[i, np.newaxis])[0] #X_test_new is the transformed X_test after PCA.\n",
    "    color = ('black' if y_pred == y_test[i] else 'red')\n",
    "    ax.set_title(y_pred, fontsize='small', color=color)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9650b9bf",
   "metadata": {},
   "source": [
    "<font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "## Performance Measures\n",
    "Let's evaluate your classifier to find out how good your model is.\n",
    "    \n",
    "    \n",
    "### Precision Recall Tradeoff (20 points)\n",
    "Plot the precision and recall versus the decision threshold figure and precision vs recall figure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddc0f2d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_recall_curve\n",
    "\n",
    "#TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb3e68a3",
   "metadata": {},
   "source": [
    "<font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "Then find the decision threshold, where the recall is equal to the precision. Write down it in the below cell:\n",
    "    <br>\n",
    "note: it should be apparent in the above figures."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10005ba5",
   "metadata": {},
   "source": [
    "decision_threshold = ?\n",
    "\n",
    "recall at the above threshold = ?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "281a0e59",
   "metadata": {},
   "source": [
    "<font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "### ROC/AUC (10 points)\n",
    "Train a RandomForestClassifier with 30 estimators then plot a ROC curve, and calculate AUC for both randomForsetClassifier and your SVM classifier that you trained earlier, which one is the best by considering AUC indicator?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "809f7ccf",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "#TODO: Train the random forest on the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39c2bb83",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve, roc_auc_score\n",
    "#TODO: plot roc_curve and print roc_auc_score for svm and random forest classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9708d110",
   "metadata": {},
   "source": [
    "<font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "### Classification Report (5 points)\n",
    "Use the <code>classification_report</code> to describe the performance of your SVM model with different metrics. Explain your results and compare the predictions of model on different classes in few sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9df2be7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "#TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "108c0ffe",
   "metadata": {},
   "source": [
    "<font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "### Confusion Matrix (15 points)\n",
    "Plot the confusion matrix of the classifier's predictions.\n",
    "Let’s focus on the errors of the plot. Divide each value in the confusion matrix by the number of images in the corresponding class so that you can compare error rates instead of absolute number of errors, then plot the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e68067ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: calculate conf_mx then print it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2832575",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: after calculating confusion matrix, visualize it using below code\n",
    "\n",
    "plt.matshow(conf_mx, cmap=plt.cm.gray)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "837c1352",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: calculate norm_conf_mx then print it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c402d54",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.matshow(norm_conf_mx, cmap=plt.cm.gray)\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
